JMD: A Hybrid Approach for Detecting Java Malware

With the rapid rise in the number of exploits targeting the Java runtime environment, new tools are required to detect these malicious Java applications. This paper proposes one such tool, the Java Malware Detector (JMD). JMD takes a hybrid approach that combines symbolic execution, instrumentation and dynamic analysis to detect malware that subverts Java’s access control mechanisms. Using this approach, we aim to derive any trigger conditions that may exist before instrumenting and executing the malware in a controlled environment to observe whether it escapes the Java security sandbox. A key element of this approach is our use of existing open-source software platforms—specifically, Java Pathfinder and AspectJ. By using real-world Java malware samples we are able to evaluate the effectiveness of JMD. The results of this evaluation show that JMD’s instrumentation and dynamic analysis capabilities provide an effective tool for detecting a wide range of Java malware: we successfully detected malware variants that represent fourteen of the known access control-related CVEs disclosed over the past four years. However, our success in using symbolic execution to derive trigger conditions was limited, mainly due to the incomplete state of the String handling implementation in Java Pathfinder’s symbolic execution plugin.